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FOREWORD 


The  SURVEILLANCE  SYSTEMS  research  program  of  the  U,  S.  Army  Behavior  and  Systems 
Research  Laboratory  has  as  its  objective  the  production  of  scientific  data  bearing  on  the 
extraction  of  information  from  surveillance  displays,  and  the  efficient  storage,  retrieval, 
and  transmission  of  this  information  within  an  advanced  computerized  interpretation 
facility.  Results  are  used  in  the  development  of  techniques  to  enhance  all  phases  of  the 
image  interpretation  process  and  to  provide  findings  for  use  in  future  systems  design. 
Research  is  conducted  under  RDT&E  Project  20662704 A721 ,  Surveillance  Systems,  FY  1971 
Work  Program. 

The  ADVANCED  SURVEILLANCE  SYSTEMS  Work  Unit  continues  BESRL  research  pre¬ 
viously  conducted  as  the  COMPONENT  INTEGRATION  Work  Unit,  with  emphasis  on  the 
human  factor  requirements  of  tactical  interpretation  facilities,  as  these  facilities  are 
currently  evolving,  particularly  with  regard  to  the  critical  analysis  and  evaluation  of  the 
functions  of  the  interpreter,  his  display  needs,  and  the  effectiveness  of  the  total  system. 
The  present  publication  summarizes  four  experiments  leading  to  more  effective  methods 
of  maintaining  the  skills  of  image  interpreters  within  an  advanced  interpretation  system 
through  feedback  furnished  by  team  members. 

BESRL  research  in  this  area  is  conducted  as  an  in-house  research  effort  augmented  by 
contracts  with  organizations  selected  as  having  unique  capabilities  and  facilities  for  re¬ 
search  in  aerial  surveillance.  The  present  research  was  conducted  jointly  by  personnel 
of  BESRL  and  of  the  System  Development  Corporation. 


TRAINING  INDIVIDUAL  IMAGE  INTERPRETERS  USING  TEAM 
CONSENSUS  FEEDBACK 


BRIEF 


Requirement: 

To  explore  the  effectiveness  of  a  low  cost  proficiency  improvement  and  maintenance 
program  for  image  interpreters;  specifically,  to  determine  the  effect  on  interpreter  detection 
and  identification  skills  of  a  training  method  using  feedback  arising  from  the  agreement 
(consensus)  which  team  members  achieve  when  they  compare  and  discuss  their  interpre¬ 
tations  , 


Procedure: 

Four  experiments  were  conducted,  exploring  various  team  consensus  feedback  methods 
with  varying  conditions  of  team  size  and  composition.  In  Experiment  I,  team  consensus  feed¬ 
back  was  compared  with  precise  feedback  (prior  identifications  made  by  experienced  inter¬ 
preters)  and  no  feedback  in  effectiveness  in  improving  both  target  detection  and  target  iden¬ 
tification.  Experiment  II  studied  the  effectiveness  of  team  consensus  feedback  in  improving 
target  detection  performance  only.  Experiment  III  evaluated  modified  team  consensus  feed¬ 
back  techniques  designed  to  minimize  feedback  delays.  In  Experiment  IV,  the  nature  of 
learning  curves  using  team  consensus  feedback  techniques  was  investigated  and  the  most 
promising  procedures  identified  In  the  first  three  experiments  were  compared  in  a  thor¬ 
ough  analysis  of  the  team  consensus  method. 


Findings: 

Team  consensus  feedback,  requiring  only  that  interpreters  compare  their  answers  in 
the  detection  and  identification  of  tactical  targets  on  previously  interpreted  or  uninterpreted 
aerial  imagery,  was  demonstrated  to  provide  greater  performance  improvement  than  a  con¬ 
trol  condition  where  interpreters  practiced  individually  with  no  feedback.  The  greatest  per¬ 
formance  gains  were  made  in  target  identification.  The  number  of  inventive  errors  was  re¬ 
duced  using  team  consensus  feedback  training;  improvement  in  the  number  of  targets  cor¬ 
rectly  detected  was  found  only  in  the  fourth  experiment  in  vvhich  the  most  promising  tech¬ 
niques  of  the  first  three  experiments  were  compared.  The  results  further  showed  that  inter¬ 
preters  who  were  initially  low  in  proficiency  achieved  the  most  significant  gains  in  performance. 

Interpreters  assigned  to  teams  that  were  heterogeneous  in  terms  of  initial  proficiency 
made  greater  gains  than  interpreters  that  were  members  of  teams  homogeneous  in  terms  of 
initial  proficiency.  Low  proficiency  interpreters  were  shown  to  make  greater  gains  in  pro¬ 
ficiency  than  interpreters  with  high  initial  proficiency.  The  results  indicate  that  low  pro¬ 
ficiency  interpreters  learned  from,  or  benefited  through  their  collaboration  with,  higher 
proficiency  interpreters. 


Team  discussion  and  team  size  did  not  significantly  influence  interpreter  performance; 
however,  there  was  some  indication  that  these  variables  combine  to  produce  an  overall  effect. 

The  greatest  performance  gains  were  achieved  with  a  method  using  three-man  teams, 
heterogeneous  in  terms  of  initial  proficiency,  in  which  individuals  made  their  initial  interpre¬ 
tations  individually  but  were  permitted  to  discuss  their  identifications  freely.  Team  methods 
requiring  the  continued  close  association  of  interpreters  from  the  beginning  of  search  did 
not  lead  to  overall  performance  improvement,  possibly  because  high  proficiency  interpre¬ 
ters  precluded  effort  on  the  part  of  less  proficient  interpreters. 


Utilization  of  Findings: 

Team  consensus  feedback  was  demonstrated  to  be  an  effective  method  of  improving 
and  maintaining  image  interpreter  proficiency  in  an  operational  image  interpretation  facility. 
The  method  requires  no  specialized  training  materials  or  instructor  preparation.  Two  or 
three  interpreters,  using  a  roll  of  tactical  imagery,  may  improve  their  performance,  parti¬ 
cularly  in  identification,  by  individually  interpreting  frames  of  imagery  and  then  discussing 
their  detections  and  identifications.  More  proficient  interpreters  should  be  included  in  each 
team  to  effect  the  greatest  improvement  in  the  less  proficient  interpreters.  This  method 
could  also  profitably  be  used  in  the  classroom  during  team  exercises. 
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BACKGROUND  AND  PURPOSE 


Context  of  the  Study 

A  series  of  experiments  has  been  conducted  over  the  past  several 
years  by  the  Behavior  and  Systems  Research  Laboratory  (BESRL)  to  develop 
and  test  the  team  consensus  feedback  method  as  a  technique  for  maintain¬ 
ing  and  enhancing  the  proficiency  of  individual  image  interpreters.  The 
essential  feature  of  the  method  is  that  interpreters  practice  in  teams, 
arriving  at  decisions  with  regard  to  target  detection  and  identification 
by  a  consensus  of  the  team  members.  Generally,  a  consensus  has  been  de¬ 
fined  as  either  a  two-thirds  vote  or  a  unanimous  vote  of  the  team  members. 
The  only  feedback  which  the  interpreters  receive  with  regard  to  the  accu¬ 
racy  of  their  decisions  they  provide  for  themselves  during  team  discus¬ 
sion  and  comparison. 

Image  interpreters  working  alone  on  a  mission  are  often  unaware  when 
they  are  doing  a  poor  job  of  detecting  or  identifying  targets.  Seldom  do 
they  receive  any  feedback,  and  if  they  do,  it  is  generally  too  late  to  be 
effective.  In  teams,  however,  interpreters  can  be  forced  continually  to 
take  stock  of  themselves,  since  their  teammates  are  finding  targets  and 
making  identifications  which  disagree  with  their  own.  In  any  conflict 
regarding  identification  of  a  specific  target,  someone  has  to  be  wrong. 
This  awareness  of  disagreement  not  only  forces  team  members  to  take  a 
long  hard  look  at  the  target,  but  also  allows  less  proficient  interpreters 
to  become  aware  of  some  of  their  own  deficiencies  and  to  learn  from  the 
more  proficient  interpreters. 

The  team  consensus  feedback  method  is  based  on  prior  team  studies 
conducted  at  BESRL  (^  ,  3),  which  demonstrated  that  image  interpreters 

working  in  teams  can  produce  more  complete  and  accurate  intelligence  in¬ 
formation  from  aerial  reconnaissance  imagery  than  interpreters  working 
alone.  The  consensual  judgment  of  team  members  was  found  to  be  espe¬ 
cially  effective  in  reducing  the  number  of  identification  errors  made  by 
single  interpreters.  Since  teams  generally  produce  better  reports  than 


^  Sadacca,  R.,  H.  Martinek,  and  A.  I.  Schwartz.  Image  Interpretation 
Task--Status  Report.  Technical  Research  Report  1129,  June  1962  (AD  411 
228).  Behavior  and  Systems  Research  Laboratory,  Arlington,  Va. 

^  Bolin,  S.  F.,  R.  Sadacca,  and  H.  Martinek.  Team  procedures  in  image 
interpretation.  Technical  Research  Note  164,  December  1965  (AD  480535) • 
Behavior  and  Systems  Research  Laboratory,  Arlington,  Va. 

^  Doten,  G.  W.,  J.  T.  Cockrell,  and  R.  Sadacca.  The  use  of  teams  in  image 
interpretation:  Infoirmation  exchange,  confidence,  and  resolving  disagree¬ 

ments.  Technical  Research  Report  1151>  October  1966  (AD  645512). 

Behavior  and  Systems  Research  Laboratory,  Arlington,  Va. 


individuals,  interpreters  training  in  teams  should  receive  more  accurate 
knowledge  of  results,  or  feedback,  than  interpreters  practicing  alone. 

The  research  approach  followed  has  been  to  evaluate  empirically 
various  team  consensus  feedback  procedures  to  determine  which  variations 
are  most  effective  in  training  individual  interpreters.  Factors  other 
than  work  procedures  which  have  been  varied  in  these  experiments  are  de¬ 
lay  of  feedback,  manner  in  which  the  feedback  is  presented,  type  of  feed¬ 
back,  size  of  team,  composition  of  the  team  in  terms  of  initial  profi¬ 
ciency  of  team  members,  and  the  effect  of  initial  proficiency  on  subse¬ 
quent  learning.  In  the  final  experiment,  reported  here  as  Experiment  IV, 
an  attempt  was  made  to  investigate  the  nature  and  extent  of  the  learning 
which  occurs  and  the  shape  of  the  learning  curve. 


OBJECTIVES 

Most  field  interpretation  units  typically  have  a  relatively  large 
number  of  inexperienced  personnel  and  a  relatively  small  number  of  exper¬ 
ienced  personnel.  Some  proficiency  improvement  and  maintenance  program 
is  necessary  for  these  personnel,  especially  for  recent  graduates  of  in¬ 
terpretation  schools  and  transferees.  The  team  consensus  feedback  method, 
if  proved  feasible,  would  offer  a  relatively  simple  and  inexpensive 
method  of  providing  this  practice.  The  advantages  of  the  method  are  that 
no  elaborate  and  expensive  materials  are  needed,  and  practice  sessions  can 
be  initiated  during  any  slack  period  by  simply  taking  a  roll  of  imagery 
"off  the  shelf"  and  presenting  it  to  interpreter  teams  in  an  image  inter¬ 
pretation  facility. 

The  four  experiments  reported  here  were  conducted  in  an  effort  to 
develop  team  practice  methods  which  will  lead  to  the  greatest  performance 
gains  by  individual  interpreters.  The  first  experiment  was  designed  to 
obtain  a  general  assessment  of  the  usefulness  of  the  consensus  feedback 
process.  In  this  experiment,  team  feedback  was  contrasted  with  no  feed-' 
back  on  the  one  hand  and  precise  feedback  on  the  other.  In  the  no-feed¬ 
back  condition,  interpreters  simply  practiced  by  themselves.  Precise 
feedback  consisted  of  the  "school  solution"  derived  by  a  team  of  expert 
interpreters  working  in  part  with  ground  survey  data.  It  was  assumed 
that  the  quality  of  team  consensus  feedback  would  lie  somewhere  between 
no  feedback  and  precise  feedback,  depending  upon  the  effectiveness  of  the 
team  procedure  being  used. 

The  second  experiment  sought  to  determine  if  team  consensus  feedback 
would  be  effective  in  enhancing  detection  performance.  In  the  first  ex¬ 
periment,  most  of  the  team  practice  had  been  concentrated  on  the  identifi¬ 
cation  of  targets.  In  the  second  experiment,  emphasis  was  shifted  simply 
by  eliminating  the  requirement  for  identification,  thereby  greatly  increas¬ 
ing  the  amount  of  team  practice  in  target  detection. 

In  the  third  experiment,  emphasis  was  concentrated  once  again  on 
target  detection,  but  team  procedures  were  changed  to  minimize  delay  of 
feedback.  This  experiment  was  also  structured  to  allow  team  output  during 
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the  practice  sessions  to  be  measured.  This  latter  step  was  taken  in 
order  to  determine  if  teams  were  detecting  more  targets  during  practice 
than  were  the  interpreters  working  alone. 

The  fourth  experiment  was  concerned  mainly  with  the  nature  of  the 
learning  whieh  occurs  during  the  team 'consensus  feedback  practice  ses¬ 
sions  and  also  with  the  shape  of  the  learning  curve.  Since  this  experi¬ 
ment  was  intended  to  be  the  final  experiment  in  the  series,  it  was  con¬ 
sidered  a  replication  of  some  of  the  preceding  experiments.  To  this  end, 
the  most  effective  procedures  identified  in  the  previous  three  experi¬ 
ments  were  evaluated. 


FRAMEWORK  FOR  THE  EXPERIMENTATION 

The  same  general  sequence  of  events  was  followed  in  all  four  experi¬ 
ments.  Each  group  of  subjects  spent  five  days  in  laboratory  experimenta¬ 
tion.  The  first  day  was  taken  up  with  instructions  and  an  individually 
administered  pre-training  test.  The  second,  third,  and  fourth  days  were 
used  for  team  practice  sessions,  and  the  fifth  day  was  used  for  the  post¬ 
training  test.  In  the  last  experiment,  a  third  individual  test  was  in¬ 
troduced  in  the  middle  of  the  third  day.  For  all  experiments,  a  control 
group  of  individual  interpreters  practiced  alone  during  the  second,  third, 
and  fourth  days  on  the  same  material  used  by  the  teams. 

The  imagery  used  in  the  experiments  consisted  of  aerial  photographs 
of  Army  field  maneuvers.  The  imagery  was  arranged  in  stereo  pairs,  and 
150  of  these  pairs  were  spliced  together  in  a  roll  of  9*'  ^  positive 
transparencies.  Each  interpreter  had  his  own  viewing  device--a  light 
table--and  other  basic  interpreter  equipment.  The  task  consisted  of 
searching  each  stereo  pair  for  designated  types  of  military  targets,  anno¬ 
tating  the  imagery  by  circling  and  numbering  the  targets,  and  then  iden¬ 
tifying  the  targets  where  identification  was  required.  The  interpreters 
then  recorded  this  information  on  data  sheets. 

The  experimental  sample  was  composed  of  enlisted  men  who  had  just 
completed  the  image  interpretation  course  at  the  U.  S.  Army  Intelligence 
School,  Fort  Holabird,  Maryland.  These  relatively  inexperienced  inter¬ 
preters  were  judged  to  have  proficiency  consonant  with  the  proficiency 
levels  of  interpreters  who  might  benefit  from  consensus  feedback  exer¬ 
cises  in  the  field.  Fifty-four  interpreters  were  used  in  the  first  exper¬ 
iment,  60  in  the  second,  48  in  the  third,  and  5^  the  final  experiment. 

FINDINGS 

Effects  of  Different  Types  of  Feedback  (Experiment  I) 

\ 

In  the  first  experiment,  team  consensus  feedback  was  compared  to  pre¬ 
cise  feedback  and  no  feedback.  All  teams  in  this  experiment  had  three 
men  assigned  to  each  team,  one  man  being  high  in  proficiency.  The  remain¬ 
ing  team  members  were  either  both  medium  in  proficiency,  b6th  low  in 
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proficiency,  or  one  medium  and  one  low*  Proficiency  was  determined  by 
means  of  the  pre-training  test*  Three  different  consensus-feedback  pro¬ 
cedures  were  used  in  this  experiment: 

Serial  Consensus*  Each  man  initially  worked  independently  on  a 
different  stereo  pair.  The  men  annotated  all  targets  found,  identified 
the  targets,  and  recorded  the  targets  on  data  forms*  The  team  members 
then  rotated  seats  and  checked  their  teammates*  work*  Checking  consisted 
of  agreeing  or  disagreeing  with  the  targets  found  by  the  previous  inter¬ 
preters  and  then  looking  for  additional  targets  missed  by  the  previous 
interpreters*  Following  checking,  the  three  men  got  together  and  dis¬ 
cussed  all  conflicts  on  each  stereo  pair* 

Immediate  Consensus*  Each  man  worked  on  a  copy  of  the  same  stereo 
pair  during  initial  interpretation.  The  men  then  went  directly  to  the 
discussion  phase  and  attempted  to  resolve  all  conflicts. 

Delayed  Consensus*  This  method  was  the  same  as  immediate  consensus 
except  that  each  man  completed  three  stereo  pairs  before  discussion  was 
begun* 

For  the  precise  feedback  condition,  two  work  procedures  were  used: 

1*  Precise  Team  Procedure --The  teams  first  went  through  the  immedi¬ 
ate  consensus  procedure  and  then  were  given  the  correct  answers  to  com¬ 
pare  with  their  team  sheet. 

2.  Precise  Individual  Procedure  - -The  men  worked  independently* 
Following  interpretation  of  each  stereo  pair,  each  interpreter  was  given 
the  correct  answers  to  compare  with  his  own* 

For  the  no-feedback  control  condition,  each  man  worked  by  himself 
and  no  feedback  was  received  at  any  time* 

In  this  first  experiment,  interpreters  practicing  under  team  consen¬ 
sus  feedback  made  greater  gains  than  interpreters  in  the  no-feedback  con¬ 
dition.  As  expected,  those  interpreters  provided  precise  feedback  showed 
the  greatest  gain.  Another  finding  indicating  the  importance  of  feedback 
was  that  the  delayed  consensus  procedure  showed  the  least  gain  among  the 
feedback  procedures.  This  finding  is  in  keeping  with  the  general  results 
in  psychological  experiments  dealing  with  delay  of  reinforcement  or  feed¬ 
back. 


A  breakdown  of  the  results  revealed  that  most  of  the  gains  in  favor 
of  the  consensus  feedback  procedures  occurred  in  identification  of  targets* 
No  significant  gains  were  obtained  in  the  number  of  targets  detected  or 
in  the  number  of  false  targets  eliminated. 
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Effects  of  Consensus  Feedback  on  Target  Detection  (Experiment  II) 

The  second  experiment  explored  the  effect  of  consensus  feedback  on 
target  detection.  The  immediate  consensus  procedure  and  the  no  feedback 
control  procedure  were  used,  but  the  requirement  to  identify  targets  was 
eliminated.  This  change  reduced  the  amount  of  time  required  for  each 
stereo  pair  and  greatly  increased  the  number  of  stereo  pairs  which  could 
be  covered  during  team  practice,  thereby  presumably  increasing  the  amount 
of  detection  practice.  Another  feature  introduced  in  the  second  experi¬ 
ment  was  the  use  of  confidence  statements.  Interpreters  were  told  that 
if  they  recorded  doubtful  targets  with  a  confidence  of  49?^  or  less,  they 
would  receive  one-half  credit  if  they  were  correct  and  would  lose  nothing 
if  they  were  incorrect.  This  technique  greatly  encouraged  responding 
during  team  practice  and  brought  many  more  doubtful  discriminations  be¬ 
fore  the  team  for  resolution.  In  addition,  this  experiment  explored  the 
effects  of  difference  in  size  of  team  (2-  and  5"i^an  teams),  the  use  of 
discussion  versus  no  discussion,  and  team  composition  based  on  whether  the 
team  members  were  hpmogeneous  or  heterogeneous  in  initial  ability. 

Team  consensus  feedback  greatly  reduced  the  number  of  inventive 
’false  alarm’  errors  hut  did  not  significantly  increase  the  number  of 
targets  correctly  detected. 

Results  with  regard  to  team  size  and  discussion  were  non-significant, 
but  results  with  regard  to  team  type  were  significant.  Interpreters  who 
trained  in  heterogeneous  teams  showed  greater  gains  than  the  homogeneous 
teams  in  both  number  of  targets  detected  and  reduction  of  inventive 
errors.  Interpreters  with  initial  low  proficiency  showed  the  greatest 
improvement  in  the  reduction  of  inventive  errors.  Proficiency  differ¬ 
ences  were  significant  for  the  reduction  of  inventive  errors  but  not  for 
the  number  of  targets  correctly  detected. 


Use  of  Joint  Search  Team  Procedures  to  Minimize  Delay  in  Consensus  Feedback 
(Experiment  III) 

The  third  experiment  explored  two  joint-search  procedures  designed 
to  minimize  the  time  interval  between  initial  detection  of  a  target  and 
team  consensus  feedback.  An  equally  important  purpose  of  the  joint- 
search  procedures  was  to  increase  the  number  of  targets  detected  by  each 
team. 


In  the  joint-search  procedure  with  the  least  delay,  each  man  had  a 
separate  light  table  and  a  copy  of  each  stereo  pair.  As  soon  as  one  man 
detected  a  target,  he  would  lean  over  and  point  to  the  target  on  the 
other  man’s  light  table.  The  men  would  then  stop  searching  and  discuss 
the  target.  After  consensus  had  been  reached,  they  would  resume  search 
until  the  next  target  was  found.  Generally,  the  first  few  easy  targets 
were  found  and  agreed  upon  in  a  very  short  timej  increasing  time  intervals 
and  longer  discussions  were  noted  as  the  targets  became  more  doubtful. 
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In  the  second  joint-search  procedure,  the  men  pointed  out  targets  to 
each  other  as  soon  as  they  were  found,  but  no  discussion  took  place  until 
both  men  had  finished  searching  the  stereo  pair.  At  this  time,  all 
located  targets  were  discussed. 

In  addition  to  these  two  work  procedures,  the  serial  search  procedure 
and  a  no-feedback  condition  used  in  the  first  experiment  were  included. 

All  teams  in  the  third  experiment  were  composed  of  two  men,  one  high 
in  initial  proficiency  and  one  low.  Confidence  estimates  and  scoring 
were  as  in  the  second  experiment.  During  the  teamwork  sessions,  half  the 
teams  were  required  to  identify  the  targets  at  a  gross  name  levelj  the 
other  half  had  only  to  detect  the  targets.  During  the  individual  pre- 
and  post-training  tests,  only  detection  was  required. 

No  significant  improvement  in  performance  was  achieved  either  in 
number  of  targets  correctly  detected  or  in  reduction  of  inventive  errors. 
Thus,  it  would  appear  that  the  notion  that  more  immediate  feedback  in¬ 
creases  learning  was  not  borne  out  by  this  experiment.  However,  practice 
decreased  the  number  of  inventive  errors  made  by  the  initially  less  pro¬ 
ficient  interpreters . 


Nature  and  Scope  of  Learning  During  Team  Consensus  Feedback  (Experiment  IV) 


In  the  first  three  experiments,  no  attempt  was  made  to  determine 
directly  if  learning  was  occurring  during  the  three  days  of  consensus 
feedback  training.  Learning  was  inferred  from  differences  between  experi¬ 
mental  and  control  groups.  In  the  final  experiment,  an  attempt  was  made 
to  determine  the  nature  and  shape  of  the  learning  curve  involved  in  team 
consensus  feedback. 

An  effort  was  made  to  choose  the  team  procedures  and  conditions 
which  had  led  to  the  largest  gains  relative  to  the  control  group  in  the 
first  three  experiments.  Two  team  procedures  used  in  Experiment  I, 
serial  consensus  and  immediate  consensus,  were  selected  as  the  most 
promising  of  the  procedures.  In  choosing  team  size,  consideration  was 
given  to  trends  and  other  indications  which  seemed  to  indicate  that  three - 
man  teams  are  preferable  to  two-men  teams.  Although  no  beneficial  effects 
had  been  demonstrated  for  discussion,  team  members  were  allowed  to  discuss 
their  target  identifications.  The  teams  were  required  to  identify  all 
targets  at  gross  name  level.  All  teams  were  heterogeneous  in  terms  of 
initial  proficiency,  each  team  consisting  of  one  high,  one  medium,  and 
one  low  proficiency  interpreter. 

Three  individual  tests  were  used  to  assess  learning;  pre-training, 
intermediate,  and  post-training.  The  three  tests  were  roughly  equated  as 
to  difficulty,  but  to  insure  that  difficulty  did  not  bear  on  the  results, 
a  Latin  square  arrangement  was  used  to  counterbalance  tests,  sessions, 
and  training  procedures. 
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Results  of  the  final  experiment  showed  very  definitely  that  learning 
occurred  during  the  three-day  practice  period  and  that  this  learning  was 
significantly  greater  for  the  team  consensus  feedback  groups.  Learning 
was  demonstrated  for  all  performance  measures--target  identification, 
number  of  targets  correctly  detected,  and  number  of  inventive  errors. 

The  immediate  consensus  feedback  procedure  seemed  slightly  superior 
to  the  serial  consensus  procedure.  With  serial  consensus  feedback  pro¬ 
cedure,  learning  was  rapid  at  first  and  then  slowed  sharply;  learning 
with  the  immediate  consensus  feedback  procedure  was  steady  at  a  medium 
rate.  For  the  control  group,  learning  occurred  at  a  low  steady  rate. 

IMPLICATION  OF  THE  FINDINGS 

The  general  conclusion  from  the  four  experiments  is  that  the  team 
consensus  feedback  is  an  effective  method  for  maintaining  and  improving 
image  interpreter  proficiency.  Although  precise  feedback  leads  to  greater 
learning,  precise  feedback  is  seldom  available,  especially  in  field  situ¬ 
ations.  Consensus  feedback  training  can  be  readily  implemented  where 
two  or  more  interpreters  have  free  time  and  a  roll  of  imagery. 

Overall  results  for  the  four  experiments  indicate  that  team  consen¬ 
sus  feedback  practice  leads  to  greater  learning  than  individual  practice' 
in  target  detection  and  identification.  The  method  has  been  shown  to  be 
most  effective  in  increasing  identification  proficiency.  The  team  con¬ 
sensus  feedback  method  was  also  effective  in  reducing  number  of  false 
targets  reported.  Only  with  the  most  effective  team  procedures  (in 
Experiment  IV)  was  there  a  significant  increase  in  number  of  targets  cor¬ 
rectly  detected. 

Team  composition  in  terms  of  proficiency  seems  highly  important  in 
the  team  consensus  feedback  procedure.  Where  teams  are  heterogeneous  in 
proficiency,  learning  is  substantially  greater  than  where  teams  are  homo¬ 
geneous  in  proficiency.  This  fact  seems  to  indicate  that  team  members 
learn  from  each  other,  and  unless  at  least  one  member  of  the  team  is  more 
proficient  than  the  others,  very  little  learning  will  occur.  This  con¬ 
clusion  was  clearly  borne  out  in  Experiment  IV  where  the  highly  profi¬ 
cient  interpreters  in  the  team  practice  groups  learned  only  a  small 
amount  more  than  the  control  interpreters,  whereas  the  interpreters  of 
medium  and  low  proficiency  learned  much  more  than  the  control  interpreters. 

The  ideal  procedure  seems  to  be  to  have  three-man  teams  whose  members 
are  heterogeneous  in  proficiency  using  the  immediate  consensus  work  proce¬ 
dure.  In  this  procedure,  all  interpreters  first  perform  initial  interprer 
tation  on  an  individual  basis  and  then  get  together  to  decide  on  the  team 
report.  The  crucial  factors  seem  to  be  the  individualized  initial  inter¬ 
pretation  and  the  opportunity  for  immediate  feedback  through  comparison 
of  responses  and  discussion  with  teammates. 
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TRAINING  INDIVIDUAL  IMAGE  INTERPRETERS  USING  TEAM  CONSENSUS  FEEDBACK 


TECHNICAL  SUPPLEMENT 
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EXPERIMENTAL  DESIGN  AND  RESULTS  OF  FOUR  EXPERIMENTS  ON  TEAM  CONSENSUS 
FEEDBACK 


METHOD 


Orientation  and  Instructions 

In  all  four  experiments,  the  first  half-day  was  taken  up  with  orien¬ 
tation  and  instructions.  Following  the  first  presentation  of  the  instruc¬ 
tions,  the  interpreters  were  given  practice  frames  containing  targets  that 
could  easily  be  identified.  Each  of  the  practice  data  sheets  was  checked 
individually  to  determine  if  the  interpreters  understood  the  instructions. 
The  instructions  were  then  repeated  for  any  procedure  which  had  been  mis¬ 
understood  by  one  of  the  interpreters.  During  this  period  and  subsequently 
throughout  the  experiments,  each  interpreter  was  given  a  set  of  photo¬ 
graphic  keys  which  contained  photographs,  scale  drawings,  and  measurements 
for  each  target  on  the  target  list.  The  photographic  keys  also  contained 
vertical  photographs  of  each  target  in  stereo  at  a  scale  similar  to  those 
used  in  the  experiments.  To  insure  that  subjects  were  thoroughly  famil¬ 
iar  with  the  keys,  an  orientation  test  consisting  of  several  frames  con¬ 
taining  annotated  targets  was  given  the  subjects.  These  orientation  tests 
contained  one  or  more  targets  for  each  type  of  target  on  the  target  list. 

To  identify  these  targets,  it  was  necessary  to  study  the  keys  thoroughly. 
Following  practice  with  the  keys,  subjects  were  given  another  orientation 
test  consisting  of  several  frames  of  imagery  similar  to  that  used  during 
the  main  experiment  to  familiarize  them  with  the  target  detection  prob¬ 
lems  they  would  encounter  during  the  pre-training  test.  During  the  in¬ 
struction  period,  no  feedback  of  any  kind  was  given  with  reference  to  any 
of  the  targets  on  the  imagery. 


Pre-training  Test 

This  test  was  given  individually  to  all  interpreters  in  all  experi¬ 
ments  on  the  second  half  of  the  first  day.  Generally,  the  test  consisted 
of  stereo  pairs  containing  a  total  of  40-75  targets.  The  stereo 

pairs  were  selected  arbitrarily  from  the  pool  of  I5O  stereo  pairs.  An 
effort  was  made  to  have  all  scales  represented  and  to  have  frames  with 
varying  numbers  of  targets.  The  tests  were  different  for  each  experiment 
although  there  was  some  overlap.  Ten  minutes  were  allowed  for  completion 
of  each  stereo  pair,  and  interpreters  were  started  and  stopped  in  unison. 
In  the  event  an  interpreter  finished  prior  to  the  end  of  the  10-minute 
period,  he  was  required  to  sit  quietly  until  the  next  stereo  pair  was 
started.  No  feedback  was  given  during  the  test,  and  interpreters  were 
not  permitted  to  talk  about  the  test  during  breaks  nor  to  compare  data 
sheets. 
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Team  Practice 


Team  practice  was  conducted  during  the  second,  third,  and  fourth 
experimental  days.  The  control  group  practiced  as  individuals  on  copies 
of  the  imagery  used  by  the  teams.  In  Experiment  I,  all  teams  were  re¬ 
quired  to  start  and  stop  each  set  of  three  stereo  pairs  at  the  same  time 
to  control  the  amount  of  material  covered  by  each  team.  During  Experi¬ 
ments  II,  III,  and  IV,  the  teams  were  allowed  to  proceed  at  their  own 
pace  to  control  the  total  amount  of  time  spent  in  practice. 

Some  of  the  work  procedures  used  by  the  teams  were  common  to  two  or 
more  experiments  and  some  were  unique.  The  procedures  were: 

1.  Serial  Consensus --This  procedure  was  used  in  Experiments  I,  III, 
and  IV.  Each  man  performs  initial  interpretation  on  a  different  stereo 
pair.  The  men  then  rotate  chairs,  correcting  and  adding  targets  to  their 
teammates*  work.  The  men  then  discuss  all  conflicts. 

2.  Immediate  Consensus --This  procedure  was  used  in  Experiments  I, 
II,  and  IV.  Each  man  performs  initial  interpretation  on  a  copy  of  the 
same  stereo  pair.  The  men  then  discuss  all  conflicts. 

5-  Delayed  Consensus --This  procedure  was  only  used  in  Experiment  I 
and  is  the  same  as  immediate  consensus  except  that  team  members  individ¬ 
ually  perform  initial  interpretation  on  three  stereo  pairs  and  then  dis¬ 
cuss  all  three  at  once. 

4.  Joint -Search --This  procedure  was  only  used  in  Experiment  III. 
Team  members  perform  initial  interpretation  together,  pointing  out  tar¬ 
gets  to  each  other  as  they  are  found.  In  one  variation,  the  teammates 
discuss  each  target  as  it  is  found;  in  the  other,  the  teammates  wait 
until  they  have  completed  the  stereo  pair  to  discuss  the  targets. 

5*  Control  Procedure --Each  man  performs  the  interpretation  by  him¬ 
self  and  does  not  discuss  his  targets  or  compare  response  sheets  with 
other  interpreters.  The  control  procedure  was  the  same  in  all  experi¬ 
ments  . 


Post- training  and  Intermediate  Tests 

A  post-training  test  was  given  in  all  experiments.  This  test  was 
given  on  the  fifth  day  and  generally  lasted  most  of  the  day.  The  test 
consisted  of  9*^7  stereo  pairs  containing  a  total  of  40-100  targets. 
The  nature  of  the  post -training  test  and  the  manner  of  administration 
were  similar  to  the  pre-training  test.  In  Experiment  IV,  an  interme¬ 
diate  test  was  given  during  the  middle  of  the  third  day. 
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Experimental  Subjects 


Image  interpreter  trainees  just  graduated  from  the  image  interpreta¬ 
tion  course  at  the  U.  S,  Army  Intelligence  School^  Fort  Holabird,  Maryland^ 
were  the  subjects  for  the  four  experiments. 


Experimental  Imagery 

The  imagery  used  in  the  experiments  were  aerial  photographs  of  Army 
field  maneuvers.  The  photos  were  arranged  in  stereo  pairs;  I50  of  these 
pairs  were  spliced  together  in  a  9**  x  9”  roll  of  positive  transparencies. 
There  was  a  stereo  overlap  of  40-60^  on  each  stereo  pair^  and  scales 
ranged  from  1:1000-1:5000.  Each  stereo  pair  contained  from  O-I9  targets. 


Confidence  Statement 

The  interpreters  in  all  four  experiments  were  required  to  place  a 
confidence  estimate  ranging  from  0-100^  after  each  target.  In  Experiment 
1,  confidence  did  not  enter  into  the  scoring^  but  in  Experiments  II,  III, 
and  IV  the  'interpreters  were  told  that  confidence  would  affect  their 
scores.  Confidence  was  only  used  with  reference  to  target  detection  and 
did  not  pertain  to  identification.  For  Experiments  II  and  III,  confi¬ 
dence  estimates  ranging  from  ^0~\00^  were  scored  with  a  weight  of  +1  for 
real  targets  and  -1  for  false  targets.  Confidence  estimates  ranging  from 
0-49^  were  scored  with  a  weight  of  +1/2  for  real  targets  and  0  for  false 
targets.  In  Experiment  IV,  an  attempt  was  made  to  expand  the  weights 
and  use  the  full  scale  of  confidence  as  follows: 


Confidence 
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-1 

-1 

-0 

-0 
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Dependent  Variables 

The  three  basic  scores  used  were  based  on  number  of  correct  identi¬ 
fications,  number  of  targets  correctly  detected,  and  number  of  inventive 
errors  (false  targets).  All  scores  were  weighted,  the  weights  being 
based  either  on  target  priority  (Experiment  l)  or  confidence  estimates 
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(Experiments  II,  III,  and  IV),  In  Experiment  I,  offensive  tracked  vehi¬ 
cles  and  artillery  were  given  greater  weight  than  other  targets,  (This 
weighting  scheme  had  little  absolute  effect  and  no  differential  effect  on 
the  results,  and  the  scheme  was  dropped  in  later  experiments.)  In  Exper¬ 
iments  II,  III,  and  IV,  weights  were  based  on  the  confidence  estimate  for 
each  target  reported.  This  weighting  scheme  also  had  little  differential 
effect  on  the  scores;  however,  it  was  retained  in  order  to  encourage  as 
much  responding  as  possible. 

The  identification  score  was  computed  by  adding  up  points  for  each 
target  correctly  identified  and  subtracting  a  point  for  each  target  incor¬ 
rectly  identified.  Targets  detected  but  not  identified  did  not  enter 
into  the  identification  score.  The  correct  detection  score  was  computed 
by  summing  points  for  each  target  on  the  target  list  correctly  detected. 
The  inventive  error  score  was  computed  by  summing  the  points  for  all 
false  targets  reported.  In  Experiment  III,  these  last  two  scores  were 
converted  to  percentage  scores  as  shown  below  to  allow  comparison  be¬ 
tween  individual  scores  and  team  score: 


Detection  completeness 


Sum  of  points  for  targets  correctly  reported 

_ X  100 _ 

Sum  of  points  for  all  possible  targets 


Detection  accuracy 


Sum  of  points  for  targets  correctly  reported 

_ X  100 _ 

Sum  of  points  for  targets  correctly 
reported  +  points  for  false  targets  reported 


EXPERIMENT  I.  EFFECTS  OF  DIFFERENT  TYPES  OF  FEEDBACK 

The  primary  objective  of  the  first  experiment  was  to  determine  the 
effectiveness  of  team  consensus  feedback  for  improving  interpreter  profi¬ 
ciency.  Team  consensus  feedback  was  compared  with  precise  feedback  and 
no  feedback.  The  experiment  involved  six  experimental  conditions: 

Team  Consensus  Feedback  Variations 


1.  Serial  consensus  feedback 

2.  Immediate  consensus  feedback 
3-  Delayed  consensus  feedback 


^  A  more  complete  description  of  this  experiment  may  be  found  in  Cockrell, 
J.  T.  Maintaining  image  interpreter  proficiency  through  team  consensus 
feedback.  Technical  Research  Note  195*  April  I968  (AD  855585) • 

Behavior  and  Systems  Research  Laboratory.  Arlington,  VA. 


14 


Precise  Feedback  Variations 


4.  Precise  team  feedback 

5.  Precise  individual  feedback 

Control  Group 

6.  No  feedback 


Experimental  Design 

The  six  experimental  groups  were  compared  by  means  of  difference 
scores  measuring  gain  in  individual  proficiency  over  pre-training  per¬ 
formance.  Nine  interpreters  were  assigned  to  each  method.  The  5^  inter¬ 
preters  were  categorized  as  to  proficiency  on  the  basis  of  their  initial 
performance  scores.  A  randomized-blocks  analysis  of  variance  was  used  to 
assess  the  significance  of  the  differences  between  initial  and  final  per¬ 
formance  scores  across  methods  and  proficiency  groups. 

The  nine  interpreters  assigned  to  each  of  the  four  team  methods  were 
formed  into  three -man  teams  of  three  types;  high-medium-medium,  high- 
medium-low,  and  high-low-low.  Assignment  of  men  at  each  proficiency  level 
to  a  team  was  on  a  random  basis.  One  high  proficiency  man  was  assigned 
to  each  team  type  on  the  hypothesis  that  less  proficient  interpreters 
would  benefit  from  working  with  more  proficient  interpreters  in  the  con¬ 
sensus  feedback  methods. 


Results  for  Experiment  I 

The  main  results  showing  the  effect  of  the  three  different  types  of 
feedback  on  interpreter  proficiency  are  presented  in  Table  1.  The  scores 
are  generally  in  the  predicted  direction;  namely,  the  precise  feedback 
methods  showed  the  greatest  overall  gain,  the  consensus  feedback  method 
showed  intermediate  gain,  and  the  no-feedback  method  showed  the  least 
gain.  Tables  2,  5;  and  4  show  the  three  sets  of  detection  and  identifi¬ 
cation  scores  arranged  according  to  proficiency  level.  Table  5  shows 
the  F-ratios  for  each  of  the  scores. 

Practice  involving  feedback  improved  only  target  identification 
significantly  and  not  detection.  Further  analysis  of  the  identification 
scores  (using  Dunnett’s  test)  indicated  that  performance  under  the  pre¬ 
cise  team  feedback  condition  was  significantly  better  than  for  the  no¬ 
feedback  condition  at  the  .01  level  and  that  performance  under  the  pre¬ 
cise  individual  feedback  and  serial  consensus  feedback  conditions  was 
better  than  that  under  the  no-feedback  condition  at  the  .05  level.  Dif¬ 
ferences  among  the  various  team  consensus  and  precise  feedback  conditions 
were  not  significant. 
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Table  1 


MEAN  DIFFERENCES  BETWEEN  PRE-TEST  AND  POST -TEST  SCORES 
FOR  THREE  MEASURES  OF  IMAGE  INTERPRETER 
PERFORMANCE --EXPERIMENT  I 


Scores 

Me thod 

Identification 

Correct 

Detection 

False  Target 
Detection* 

Precise  Feedback 

51.2 

75.9 

21.6 

Consensus  Feedback 

41.1 

71.9 

19.1 

Control 

28.0 

69.4 

15.1 

6 

Positive  velue  represents  e  decreese  In  number  of  false  terget  responses. 


EXPERIMENT  II.  EFFECTS  OF  CONSENSUS  FEEDBACK 
ON  TARGET  DETECTION  ^ 


The  primary  objective  of  the  second  experiment  was  to  determine  the 
effect  of  team  consensus  feedback  on  target  detection.  In  Experiment  I, 
there  was  no  significant  difference  between  the  control  group  and  experi¬ 
mental  groups  on  the  detection  skill.  It  was  felt  that  this  may  have 
been  due  to  the  lack  of  sufficient  practice  on  this  skill.  Accordingly, 
in  Experiment  II,  the  requirement  for  target  identification  was  elimi¬ 
nated,  and  interpreters  were  required  only  to  indicate  the  location  of 
each  target  and  their  confidence  that  it  was  truly  a  target.  Each  team 
in  Experiment  II  completed  approximately  four  times  as  many  practice 
stereo  pairs  as  the  teams  in  Experiment  I  and  thereby  presumably  in¬ 
creased  amount  of  detection  practice. 


^  A  more  complete  description  of  this  experiment  may  be  found  in 
Cockrell,  J.  T.  Maintaining  Target  Detection  Proficiency  Through 
Team  Consensus  Feedback.  Technical  Research  Note  219 .  October  19^9 
(ad  70737^)*  Behavior  and  Systems  Research  Laboratory.  Arlington,  VA. 
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MEAN  PERFC«MANCE  GAINS  IN  IDENTIFICATION  SCORE  BETWEEN 
PRE-TEST  AND  POST -TEST --EXPERIMENT  I 
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Table  5 


F-RATIOS  FOR  DIFI  RENCE  SCORES --EXPERIMENT  I 


Score 

Source 

df 

Identifica¬ 

tion 

Correct 

Detection 

False  Target 
Detection 

Methods  (m) 

5 

2.64* 

1.33 

1.25 

Proficiency  (P) 

2 

.29 

.86 

.31 

MP 

10 

1.41 

1.75 

.80 

Within 

(Mean  Square) 

26 

349.1 

242.1 

276.7 

Total 

53 

*P  <  X)5. 


Experimental  Design 

In  Experiment  II,  in  addition  to  a  detailed  study  of  the  effects  of 
team  consensus  feedback  on  target  detection,  four  other  factors  were 
studied:  l)  team  size  (2-man  or  3“nisti),2)  team  composition  (team  members 

homogeneous  in  initial  ability  versus  heterogeneous),  3)  discussion 
versus  no  discussion,  and  4)  interpreter  proficiency  (high,  medium,  and 
low).  The  design  used  for  this  experiment  was  a  2x2x2x3  factorial  with 
a  control  group.  The  detailed  breakdown  is  shown  in  Table  6. 

Performance  among  the  eight  experimental  groups  and  the  no-feedback 
(control)  group,  shown  in  the  columns  in  Table  6,  was  compared  by  means 
of  difference  scores  obtained  by  subtracting  pre-training  test  scores 
from  post-training  test  scores.  A  randomized -blocks  analysis  of  variance 
was  used  to  assess  the  significance  of  differences  among  the  experimental 
conditions.  The  three  blocks  were  based  on  initial  proficiency  as  mea¬ 
sured  by  the  pre-training  test.  From  each  block,  two  men  were  drawn 
randomly  for  each  of  the  experimental  groups.  The  six  interpreters 
assigned  to  each  of  the  eight  experimental  groups  were  assigned  to  homo¬ 
geneous  or  heterogeneous  teams  and  to  two-man  or  three -man  teams  as 
shown  in  Table  7*  The  control  group  included  12  interpreters. 
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EXPERIMENTAL  DESIGN  FOR  EXPERIMENT  II 
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Table  7 


COMPOSITION  BY 

PROFICIENCY  OF  TEAMS 

IN  EXPERIMENT  II 

Team  Type 

3 -Man  Teams 

2-Man  Teams 

1. 

High,  High,  Medium 

1. 

High,  High 

Homogeneous 

2. 

Medium,  Low,  Low 

2. 

Medium,  Medium 

3. 

Low,  Low 

1. 

High,  Medium,  Low 

1. 

High,  Low 

Heterogeneous 

2. 

High,  Medium,  Low 

2. 

High,  Medium 

3 .  Medium^  Low 


Team  Procedures 

Team  procedures  used  in  Experiment  II  were  two  variations  on  the 
immediate  consensus  procedure.  Where  discussion  was  used^  the  procedure 
was  the  same  as  in  Experiment  I.  Where  discussion  was  not  used^  the 
teammates  merely  passed  their  completed  data  forms  to  each  other  so  that 
comparisons  could  be  made. 

Results  for  Experiment  II 

The  results  for  Experiment  II  were  analyzed  in  two  steps.  The  first 
step  compared  the  differences  among  the  various  experimental  factors;  the 
second  step  compared  the  team  consensus  feedback  groups  to  the  control 
gi^oup.  Table  8  shows  the  mean  performance  gains  for  the  various  factors 
in  terms  of  scores  based  on  number  of  correct  detections  and  number  of 
inventive  errors.  This  table  shows  a  substantial  difference  in  favor  of 
heterogeneous  teams  for  both  scores^  and  a  large  reduction  in  the  number 
of  inventive  errors  by  low  and  medium  proficiency  interpreters. 

Table  9  presents  the  analysis  of  variance  results  for  the  four 
factors  considered.  Team  size  and  discussion  versus  no-discussion  com¬ 
parisons  showed  no  significant  performance  increases.  Interpreter  pro¬ 
ficiency  (high,  medium,  and  low)  showed  a  significant  difference  in  the 
reduction  of  inventive  errors;  low  proficiency  interpreters  showed  a 
marked  reduction  in  inventive  errors,  while  more  proficient  interpreters 
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showed  less  marked  improvement.  The  performance  of  heterogeneous  teams 
improved  significantly  more  than  that  of  homogeneous  teams  both  in  number 
of  targets  correctly  detected  and  number  of  inventive  errors.  From  these 
last  two  sets  of  results,  there  is  the  indication  that  low  proficiency 
interpreters  gain  from  the  team  consensus  feedback  method  and  that  they 
benefit  from  their  association  with  a  more  proficient  team  member. 

In  order  to  analyze  the  difference  between  the  no-feedback  control 
group  and  the  consensus  feedback  experimental  groups,  the  discussion  and 
team-size  factors  were  collapsed,  leaving  only  team  type  and  proficiency 
factors.  Table  10  shows  the  mean  difference  scores  for  this  analysis 
in  both  detection  scores.  Table  11  shows  the  analysis  of  variance  for 
groups  with  unequal  numbers  of  subjects.  Team  consensus  feedback  train¬ 
ing  resulted  in  a  significant  reduction  in  the  number  of  inventive  errors^ 
when  compared  with  performance  of  the  control  group.  Team  consensus 
feedback,  however,  did  not  improve  detection  performance  in  terms  of 
number  of  targets  detected.  The  results  in  Table  11  also  repeat  the 
finding  of  significant  gains  in  error  reduction  for  the  low  proficiency 
interpreters . 


EXPERIMENT  III.  JOINT  SEARCH  PROCEDURES  TO  MINIMIZE 
DELAY  IN  CONSENSUS  FEEDBACK 

In  Experiment  III,  two  new  teamwork  procedures  were  designed  to 
minimize  the  time  delay  between  initial  interpretation  and  consensus 
feedback.  These  procedures  used  joint  target  search  rather  than  individ¬ 
ual  search.  A  reduction  in  feedback  time  delay  was  felt  to  be  important 
in  view  of  the  results  from  Experiment  I  with  delayed  consensus  feedback. 


Team  Methods 

The  two-man  team  procedures  were  as  follows: 

Joint  Search  --  One-Response  Delay.  Interpreters  search  together 
and  discuss  each  target  as  it  is  detected. 

Joint  Search  --  One-Frame  Delay.  Interpreters  search  together  but 
discuss  all  targets  at  end  of  frame. 

Serial  Search  --  Two-Frame  Delay.  Interpreters  search  individually 
on  different  frames,  cross-check  by  swapping  seats,  then  discuss  each 
frame  in  turn. 

Independent  Search  --  No  Feedback.  In  this  control  group,  inter¬ 
preters  search  individually  with  no  feedback  at  any  time. 

During  team  practice  sessions,  one  half  the  teams  were  required  to 
make  a  gross  identification  of  all  targets  (e.g.,  tracked  vehicle,  tTuck, 
shelter).  This  gross  identification  was  not  required  during  the  pre¬ 
training  and  post-training  tests.  The  purpose  of  the  identification 
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Table  8 


MEAN  GAINS  BETWEEN  PBIE-TEST  AND  POST-TEST 
IN  DETECTION  SCORES- -EXPERIMENT  II 


Experimental 

Correct 

False  Target 

Factors 

Detection 

Detection 

Team  Type 

Heterogeneous 

66.4* 

9.7* 

Homogeneous 

56.3 

0.9 

Discussion 

Yes 

61.0 

4.1 

No 

61.8 

6.6 

Team  Size 

3 -Man 

62.8 

7.9 

2-Man 

60.0 

2.8 

Initial  Proficiency 

High 

66.4 

0.2** 

Medium 

60.9 

9.2 

Low 

56.9 

19.8** 

•P  <  ^5. 

••P  <  .01. 
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Table  9 


DIFFERENCE  SCORE  F- RATIOS  FOR  DETECTION  SCORES- -EXPERIMENT  II 


F-Ratios 

Source 

df 

Correct  False  Target 

Detection  Detection 

Team  Type  (T) 

1 

4.34* 

6.89* 

Proficiency  (P) 

2 

1.28 

12.37** 

Discussion  (D) 

1 

.02 

.55 

Team  Size  (S) 

1 

.31 

2.36 

TP 

2 

.68 

.17 

TD 

1 

.41 

.63 

TS 

1 

.61 

.80 

PD 

2 

.42 

.45 

PS 

2 

.08 

3.26 

DS 

1 

.56 

.01 

TPD 

1 

.05 

.35 

TPS 

2 

.01 

1.08 

TDS 

1 

.01 

.55 

PDS 

2 

3.39 

4.23* 

TPDS 

2 

.40 

.12 

Within 

(Mean  Square) 

2A 

283.4 

135.8 

Total 

47 

•P  <  X)6. 
••P  <  X)1. 
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Table  10 


CCMPARISONS  BETWEEN  EXPERIMENTAL  GROUPS  AND  CONTROL  GROUPS  FOR 
MEAN  GAINS  IN  DETECTION  SCCttlES- -EXPERIMENT  II 


Correct  Detection 

Proficiency 

Group 

Heterogeneous 

Teams 

Homogeneous 

Teams 

Control 

High 

67.8 

65.0 

71.7 

Medium 

66.5 

55.4 

70.0 

Low 

65.1 

48.6 

65.5 

All  Groups 

66.5 

56.3 

69.1 

False  Target  Detection* 

Heterogeneous 

Teams 

Homogeneous 

Teams 

Control 

High 

0.2 

-8.0 

-29.0 

Medium 

9.2 

-2.2 

-23.0 

Low 

19.8 

13.0 

5.0 

All  Groups 

9.8 

-0.9 

-19.2 

a 

Positive  score  represents  decreese  In  the  number  of  felse  target  responses. 


Table  11 

F-RATIOS  FOR  COMPARISON  BETWEEN  COLLAPSED  EXPERIMENTAL  GROUPS 
AND  CONTROL  GROUPS  IN  DETECTION  SCC«ES- -EXPERIMENT  II 


F- 

■Ratios 

Source 

df 

Correct 

Detection 

False  Target 
Detection 

Method  (M) 

2 

3.01 

12.12** 

Proficiency  (P) 

2 

1.18 

6.73* 

M  X  P 

4 

.30 

.07 

Within 

(Mean  Square) 

51 

270.6 

322.2 

Total 

59 

•P  <  /)5. 

••P  <  .01. 
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requirement  was  to  determine  if  team  output  would  be  affected  in  terms 
of  amount  and  accuracy.  A  count  of  the  number  of  stereo  pairs  completed 
during  the  teamwork  sessions  showed  that  the  identification  requirement 
had  no  effect  on  amount  of  interpretation  accomplished. 


Experimental  Design 

For  Experiment  III,  the  variables  studied  were  team  method,  require¬ 
ment  for  identification  (required  versus  not  required),  and  initial  pro¬ 
ficiency  (high  versus  low).  The  experimental  design  is  a  4x2x2  factorial 
with  pre-post  test  difference  scores  as  dependent  variable-  Table  12 
shows  a  detailed  breakdown. 


Results  for  Experiment  111 

While  it  could  be  expected  from  the  rationale  for  the  team  consensus 
feedback  method  that  variations  providing  more  immediate  feedback--such 
as  the  joint-search  methods --would  result  in  improved  performance,  this 
notion  was  not  borne  out  by  the  results.  Tables  15  and  14  show  the  pre¬ 
test,  post-test,  and  difference  percentage  scores  based  on  number  of 
correct  detections  and  number  of  inventive  errors,  respectively.  Differ¬ 
ence  scores  between  initial  and  final  proficiency--in  terms  of  mean  per¬ 
centage  gain--were  used  for  the  analysis  of  means,  the  results  of  which 
are  shown  in  Table  15-  There  were  no  significant  differences  for  feedback 
method,  i.e.,  no  demonstrated  superiority  of  the  more  immediate  feedback 
methods  over  other  feedback  methods.  The  only  significant  difference 
found  was  for  initial  proficiency;  low  proficiency  interpreters  showed 
significant  reduction  in  the  number  of  inventive  errors  made. 


EXPERIMENT  IV.  NATURE  OF  LEARNING  DURING  TEAM 
CONSENSUS  FEEDBACK 

The  purpose  of  Experiment  IV  was  to  determine  the  nature  of  the 
learning  which  occurs  during  the  teamwork  sessions,  and  to  some  extent, 
the  shape  of  the  learning  curve.  This  experiment  differed  from  the  pre¬ 
ceding  experiments  in  that  it  permitted  measurement  of  amount  of  learning 
independent  of  test  difficulty.  Three  proficiency  tests  were  used  which 
were  counterbalanced  to  eliminate  the  effects  of  test  difficulty. 

The  procedures  and  team  composition  which  had  been  found  to  be  best 
in  previous  experiments  were  used  --  three -man  teams,  heterogeneous  teams 
in  terms  of  initial  ability,  and  discussion  as  team  members  wished.  The 
work  procedures  were  serial  consensus  and  immediate  consensus  plus  a 
control  group. 
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Table  13 


MEAN  PERCENTAGE  SCORES  FOR  DETECTION  COMPLETENESS 
ON  PRE-TEST  AND  POST -TEST --EXPERIMENT  III 


Proficiency  Tests 

Factors 

Initial  Test  Final  Test  Difference 

Team  Method 


Joint-Search-One  Response 

42.9 

51.0 

8.1 

Joint-Search-One  Pair 

42.5 

54.3 

11.8 

Serial  Search-Two  Pairs 

43.7 

52.7 

9.0 

Independent  Search 

42.6 

53.4 

10.8 

Identification 

Yes 

43.1 

54.4 

11.3 

No 

42.8 

51.3 

8.5 

Initial  Proficiency 

High 

48.5 

57.6 

9.1 

Low 

37.3 

48.1 

10.8 
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Table  14 


MEAN  PERCENTAGE  SCORES  FOR  DETECTION  ACCURACY  ON 
PRE-TEST  AND  POST-TEST- -EXPERIMENT  III 


Proficiency  Tests 

Factors 

Initial  Test  Final  Test 

Difference 

Team  Method 

Joint-Search-One  Response 

83.4 

91.4 

8.0 

Joint-Search-One  Pair 

83.5 

86.0 

2.5 

Serial  Search 

81.8 

81.3 

-0.5 

Independent  Search 

81.7 

83.9 

2.2 

Identification 

Yes 

82.8 

85.2 

2.4 

No 

82.4 

86.1 

3.7 

Initial  Proficiency 

High 

92.6 

87.4 

-5.2 

Low 

72.6 

84.0 

11.4 
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Table  15 


F-RATIOS  FOR  DIFFERENCE  SCORES  BETWEEN  PRE-TESTS 
AND  POST -TEST --EXPERIMENT  III 


Source 

df 

Detection 

Completeness 

Detection 

Accuracy 

Team  Method  (M) 

3 

.77 

.63 

Identification  (I) 

1 

2.02 

.09 

Initial  Proficiency  (P) 

1 

.69 

13.39** 

MI 

3 

2.84 

.10 

MP 

3 

1.19 

1.32 

IP 

1 

.40 

1.63 

MIP 

3 

.44 

.53 

Within 

(Mean  Square) 

32 

49.93 

246.15 

Total 

47 

••p  <  .01. 
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Experimental  Design 


Each  subject  was  given  one  of  the  three  individual  performance  tests 
as  a  pre-training  test.  Based  on  scores  on  this  test,  interpreters  were 
divided  into  three  proficiency  blocks  --  high,  medium,  and  low.  One  man 
was  randomly  drawn  from  each  block  to  form  three-man  teams  and  these 
teams  were  assigned  randomly  to  one  of  the  three  work  procedures.  Table 
16  shows  the  design  for  the  experiment. 


Results 

Results  for  Experiment  IV  are  summarized  in  Table  17*  In  general, 
improved  performance  was  found  across  the  three  sessions,  including  that 
for  the  no-feedback  (control)  group.  However,  the  two  consensus  feedback 
groups  showed  greater  learning  than  the  control  group  on  all  three  per¬ 
formance  measures  --  identification,  correct  target  detection,  and  inven¬ 
tive  errors.  Thus,  the  two  consensus  feedback  methods  were  effective  in 
improving  interpreter  proficiency.  Figures  1,  2,  and  5  present  a  break¬ 
down  of  results  by  proficiency  level.  For  all  three  performance  vari¬ 
ables,  the  greatest  performance  gains  were  achieved  by  low  proficiency 
interpreters.  Medium  proficiency  interpreters  gained  less,  and  high  pro¬ 
ficiency  interpreters  gained  little  more  than  interpreters  in  the  control 
(no  feedback)  condition. 

Table  l8  shows  the  results  of  the  analysis  of  variance  for  factors 
of  primary  experimental  concern  --  feedback  method,  initial  proficiency, 
and  sessions  --  as  well  as  for  order  and  test.  The  analysis  of  variance 
was  carried  out  on  performance  score  (not  performance  gain  as  in  prior 
experiments)  for  identification,  correct  detections,  and  inventive  errors. 

The  results  presented  in  Table  l8  show  clearly  that  (initial)  proficiency 
is  an  important  factor,  the  low  proficiency  interpreters  benefiting  most 
from  team  consensus  feedback  training.  The  results  also  show  that  learn¬ 
ing  occurred,  as  evidenced  by  the  significant  sessions  effect. 

The  apparent  lack  of  significant  results  for  method  (except  for  the 
correct  detection  score)  requires  a  word  of  explanation.  As  subjects 
were  assigned  to  experimental  methods  on  the  basis  of  initial  test  scores, 
differences  among  methods  across  the  three  tests  (pre-test,  intermediate 
test,  and  post-test)  would  be  reduced.  For  this  reason,  additional  anal¬ 
yses  were  carried  out  to  look  in  more  detail  at  differences  among  feed¬ 
back  methods.  The  Newman-Keuls  test  (S)  was  used  to  compare  the  two  feedback 
methods  against  the  control  group  and  against  each  other  for  each  of  the 
three  tests  separately  as  shown  in  Table  19*  The  application  of  the 
Newman-Keuls  test  to  scores  on  the  pre-test  shows  that,  as  desired,  there 
were  no  significant  differences  among  the  the  two  feedback  conditions  and 


Winer,  B.  J.  Statistical  Principles  in  Experimental  Design,  McGraw 
Hill  Book  Company,  New  York,  1962. 
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Table  16 


EXPERIMENTAL  ARRANGEMENT  FOR  EXPERIMENT  IV 
(N  =  5^  Subjects,  2  Per  Cell) 


Method 

Order 

Proficiency 

1 

Sessions 

2 

5 

I 

H 

M 

Test  A 

Test  B 

Test  C 

Immediate  Consensus 

II 

L 

H 

M 

B 

C 

A 

III 

L 

H 

M 

C 

A 

B 

L 

I 

H 

M 

A 

B 

C 

Serial  Consensus 

II 

L 

H 

M 

B 

C 

A 

III 

L 

H 

M 

C 

A 

B 

L 

I 

H 

M 

A 

B 

C 

Control 

II 

L 

H 

M 

B 

C 

A 

III 

L 

H 

M 

C 

A 

B 

L 
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the  control  condition  and  between  the  feedback  conditions,  since  feed¬ 
back  was  not  introduced  in  the  pre-test  condition.  However,  Table  I9 
does  show  significant  differences  between  both  feedback  groups  and  the 
control  group  in  both  identification  and  correct  detections  in  subse¬ 
quent  sessions.  No  significant  differences  were  found  for  inventive 
errors.  These  results,  then,  again  show  that  team  consensus  feedback 
is  effective  in  achieving  increased  interpretation  proficiency,  particu¬ 
larly  in  target  identification. 


Table  17 

MEAN  SCORES  FOR  IDENTIFICATION,  CCRRECT  DETECTION,  AND  INVENTIVE 
ERRORS  ACROSS  TEST  ADMINISTRATIONS- -EXPERIMENT  IV 


Dependent  Test  Administration 


Var  iab  le 

Method 

1 

2 

3 

Identification 

Immediate  Consensus 

52.1 

68.8 

76.9 

Serial  Consensus 

49.8 

71.5 

74.7 

Control 

50.6 

59.9 

66.7 

Average 

50.8 

66.7 

72.8 

Correct  Detection 

Immediate  Consensus 

86.6 

102.3 

117.6 

Serial  Consensus 

81.7 

115.5 

116.9 

Control 

83.1 

93.2 

101.5 

Average 

83.8 

103.7 

112.0 

Inventive  Errors 

Immediate  Consensus 

-16.3 

-11.0 

-9.1 

Serial  Consensus 

-15.8 

-16.8 

-13.5 

Control 

-13.9 

-16.0 

-14.9 

Average 

-15.3 

-14.6 

-12.5 
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Table  18 

ANALYSIS  OF  VARIANCE  OF  IDENTIFICATION,  POSITIVE  DETECTION, 
AND  FALSE  TARGET  DETECTION- -EXPERIMENT  IV 


F-Ratios 


Source  of  Variance 

df 

Identification 

Correct 

Detection 

False  Target 
Detection 

Between  Subjects 

Method  (M) 

2 

3.25 

3.41* 

1.79 

Order  (0) 

2 

12.17** 

5.90** 

5.23* 

Proficiency  (P) 

3 

17.36** 

15.83* 

4.05* 

MO 

4 

1.04 

1.90 

2.50 

MP 

4 

.81 

.78 

.98 

OP 

4 

1.42 

.48 

.76 

MOP 

4 

.72 

.96 

.71 

Subjects  Within 
Groups  (Mean  Sq.) 

27 

239.08 

648.41 

93.07 

Within  Subjects 

Sessions  (S) 

2 

113.70** 

60.00 

3.48* 

Test  (T) 

2 

.41 

1.88 

.44 

MS 

4 

3.50* 

4.39** 

3.07* 

MT 

4 

.44 

1.94 

4.68** 

PS 

4 

6.77 

2.35 

9.94** 

PT 

4 

.67 

.86 

1.46 

MPS 

8 

.46 

1.36 

.86 

MPT 

8 

1.45 

1.04 

.49 

Residual 

18 

2.24* 

.64 

1.58 

Error  (Mean  Sq.) 

54 

61.12 

189.03 

34.12. 

Total 

108 

•p  <  ^5. 

•  •  p  <  .01. 
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IDENTIFICATION  SCORE  IDENTIFICATION  SCORE  IDENTIFICATION  SCORE 


IDENTIFICATION  SCORE 


100 


80 


60 


^  SERIAL 
CONSENSUS 


IMMEDIATE 

CONSENSUS 


- ^  CONTROL 


I  2  3 

SESSIONS 


SESSIONS 


Figure  1.  Identification  learning  curves  for  three  methods  used  by  interpreters  of  high, 
medium,  and  low  proficiency. 
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DETECTION  SCORE  DETECTION  SCORE  DETECTION  SCORE 


CORRECT  DETECTION  SCORE 


140 


^  SERIAL 
CONSENSUS 


IMMEDIATE 

CONSENSUS 


- ^  CONTROL 


2  3 

SESSIONS 


Figure  2.  Detection  skill  learning  curves  for  three  methods  used  by  interpreters  of  high, 
medium,  and  low  proficiency 
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INVENTIVE  ERROR  SCORE 


SERIAL 

CONSENSUS 

IMMEDIATE 

CONSENSUS 

CONTROL 


Figure  3.  Inventive  error  learning  curves  for  three  methods  used  by  interpreters  of  high, 
medium,  and  low  proficiency. 
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Table  19 


P-VALUES  FOR  THE  NEWMAN  KEULS-TEST  FOR  THE  THREE  SESSIONS 
SEPARATELY  FOR  ALL  SCORES- -EXPERIMENT  IV 


Session 


Score 

Comparison 

1 

2 

3 

IC  vs  Control 

ns 

.05 

.05 

Identification 

SC  vs  Control 

ns 

.05 

.05 

SC  vs  IC 

ns 

ns 

ns 

IC 

vs 

Control 

ns 

ns 

.05 

Correct  Detection 

sc 

VS 

Control 

ns 

.01 

'  .05 

sc 

VS 

IC 

ns 

.05 

ns 

IC 

vs 

Control 

ns 

ns 

ns 

False  Target  Detection 

SC 

vs 

Control 

ns 

ns 

ns 

SC 

vs 

IC 

ns 

ns 

ns 
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APPENDIX  A 


TARGET  LIST 


T  TRACKED  VEHICLES 
TT  Tanks 

TS  SP  (Guns,  Howitzers,  Mortars,  Anti-aircraft) 

TA  APC  *  s 

TB  Armored  Bridge  Launchers 
TR  Recovery  Vehicles 
TP  Prime  Mover/Tractor 

A  ARTILLERY 

AT  Towed  Howitzers 
AM  Mortar 
AA  Anti-aircraft 
AK  Anti-tank 

M  MISSILES 

MS  Surface-to-Surface  Missile 
ML  Missile  Launcher/Transporter 
MT  Missile  Transporter 
MA  SAM 

MM  SAM  Launcher/Transporter 

W  WHEELED  AND  CONSTRUCTION  VEHICLES 

WL  Light  Cargo  Trucks,  1/4  Ton,  5/4:  Ton,  Ambulance 
WH  Heavy  Cargo  Trucks,  2-1/2  Ton,  5  Ton,  10  Ton 
WK  Tank  Trucks  (Water,  Fuel) 

WW  Wrecker  Trucks 

WT  Truck  Tractor  (List  Separate  from  Trailer) 

WV  Van  Trucks  (Generator,  Shop,  Communication,  Radar) 

WD  Dump  Truck 

WC  Construction  Vehicles  (Bulldozers,  Cranes,  Shovels,  Scoops,  etc.) 

L  TRAILERS  (ANNOTATE  SEPARATELY  FROM  TRUCKS  EVEN  IF  ATTACHED) 

LL  Light  Cargo,  1/4  Ton,  5/4=  Ton 
LH  Heavy  Cargo,  1-1/2  Ton 

LS  Small  Special  Purpose  (Ammo,  Generator,  Water,  Fuel) 

LR  Large  Special  Purpose  (Lo  Boy,  Tank  Transporter,  Van,  Tanker) 

LE  House  Trailers  (Military) 

C  CANVAS  SHELTER 

CS  Small  Personnel  Tents  (Pup,  Wall) 

CM  Medium  Special  Purpose  Tents  (CP,  Hex,  Kitchen) 

CL  Large  Tents  (GP,  Maintenance,  Hospital) 

CC  Miscellaneous  (Latrine,  Canvas  Shelter,  Canvas  Water  Tank, 

Canvas  Covered  Supplies,  Canvas  Covered  Garbage  Pits,  Flys) 
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DISTRIBUTION 
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13.  AeSTRACT 


In  a  series  of  experiments  conducted  over  the  past  several  years,  the  ADVANCED 
SURVEILLANCE  SYSTEMS  Work  Unit  of  the  Behavior  and  Systems  Research  Laboratory  ( BESRL) 
has  sought  to  develop  and  test  the  team  consensus  feedback  method  as  a  technique  for 
maintaining  and  enhancing  the  proficiency  of  individual  image  interpreters.  In  the  pres¬ 
ent  research,  conducted  jointly  by  the  BESRL  unit  and  personnel  of  the  System  Develop¬ 
ment  Corporation,  four  experiments  were  undertaken  to  explore  various  team  consensus 
feedback  methods  with  varying  conditions  of  team  size  and  composition.  Specifically, 
the  experiments  were  directed  to  developing  team  practice  methods  which  would  lead  to 
the  greatest  performance  gains  by  individual  interpreters  in  an  advanced  interpretation 
system. 

The  first  experiment  was  designed  to  obtain  a  general  assessment  of  the  usefulness 
of  the  consensus  feedback  process  in  both  target  detection  and  target  identification. 
The  second  experiment  sought  to  determine  if  team  consensus  feedback  would  be  effective 
in  enhancing  target  detection  performance  only.  Experiment  III  evaluated  modified  team 
consensus  feedback  techniques  designed  to  minimize  feedback  delays.  Experiment  IV  was 
concerned  mainly  with  the  nature  of  learning  curves  using  team  consensus  feedback  tech¬ 
niques  and  with  evaluation  of  the  most  effective  procedures  identified  in  the  first 
three  experiments. 

Team  consensus  feedback  was  demonstrated  to  be  an  effective  method  of  maintaining 
and  improving  image  interpreter  proficiency  in  an  operational  image  interpretation 
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15*  'ABSTRACT  continued 

showed  that  interpreters  who  were  initially  low  in  proficiency  achieved 
the  most  significant  gains  in  performance.  Low  proficiency  interpreters 
learned  from,  or  benefited  through  their  collaboration  with,  higher  pro¬ 
ficiency  interpreters.  Neither  team  discussion  nor  team  size  significantly 
affected  interpreter  performance;  however,  these  variables  did  combine  to 
produce  an  overall  effect.  The  ideal  procedure  seems  to  be  one  using 
three-man  teams,  heterogeneous  in  terms  of  initial  proficiency,  in  which 
individuals  first  perform  initial  interpretation  on  an  individual  basis 
and  then  discuss  their  identifications  freely  in  deciding  on  the  team 
report . 
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